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why do we still want to be subjugated? 
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Abstract 

A collective obsession with the Impact Factor (IF) has led to major changes in political science in many 
countries, and in recent years, these changes have been especially visible in Brazil. Despite critiques of both 
this measure’s ability to evaluate the scientific publications and its spread in the evaluation of researchers and 
institutions, Brazil has taken the IF as an important element in evaluating Brazilian graduate programs. In this 
article, we briefly try to demonstrate that the IF can be subject to many biases that seem to be completely 
ignored or unknown. 
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Rcsumo 

A obscssao pelo Fator de Impacto (FI) tern lcvado a grandes mudan^as nas politicas cicntificas de muitos 
paiscs, cspecialmcnte nos ultimos anos no Brasil. Apesar de todas as criticas c discordances dcssa medida para 
avaliar as publicans cicntificas, c a sua cxtensSo na avalia<?ao de pesquisadores c institutes, o Brasil tomou 
o FI como um importantc elemento para avaliar a pds-gradua^So brasileira. Neste artigo de opiniao, procurou- 
sc demonstrar brevemente que o FI pode ser uma fontc de muitos vicscs que parcccm ser complctamcnte 
ignorados ou desconhecidos. 

Palavras-chave: tHica cm cieneia, politiea editorial, taxa de citato, vieses de publica^ao. 


Introduction 

We have certainly witnessed a great change 


in the way that we regard scientists and, indeed, 
practice science itself. Though these questions arc 
perhaps as old as scientific debate, the recurrence 
with which some topics arise throughout history is 
truly impressive. Without a doubt, one of these 
important recurring topics is the evaluation of 
scientists by the Impact Factor of the journals in 
which they publish their work (see Colquhoun 2003). 

In recent years, especially in Brazil, the Impact 
Factor of journals has received considerable 
attention, because it is one of the criteria used to 
evaluate the quality of graduate courses in Brazil, 
and therefore, to evaluate scientists. However, it is 
necessary to emphasize that the IF docs not 


evaluate the quality or output of scientific work, 
but only its visibility. While there is certainly no 
consensus on this subject, the IF is supported by a 
great majority of scientists; unfortunately, it is not 
clear to young students and new researchers 
whether an evaluative policy for staff members wall 
be adopted based on the Impact Factor. 
Surprisingly, despite all of its shortcomings, many 
of which have been previously elaborated upon 
(e.g. Kokko & Sutherland 1999), the IF is 
increasingly used to evaluate institutions and to 
determine how resources will be allocated (see Ruiz 
etai 2009). 

According to Kokko & Sutherland (1999:382) 
“the impact factor is the average number of times 
that articles published in a specific journal in the 
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Box 1 - Questions that 
may be answered by the 
impact factor. Text 
reproduced and modified 
for the box format from 
Dong etal. (2005). 


What question does the impact factor answer? 

Strictly speaking, the journal IF only measures the average citation rate of ail the “citable” 


articles (research articles, technical notes and reviews) in a journal. As such, IF is not a 
perfect tool to measure the journal quality. However, in most cases, it performs what it 
promises when various flaws are taken into active consideration. Ready accessibility and 
regular updates of the 1S1 IF provides the best available indicator for journal quality, 
accepted widely w r ithin the scientific community. 


What questions does the impact factor not answer? 

The IF cannot assess the quality of individual articles, due to the qualitative variety of 
citations distributed in a journal. A small proportion of articles count fora large percentage 
of citations. This means that a typical article in a high IF journal may not be cited more 
frequently than an average article in a relative low IF journal. As a result, IF alone is not 
able to judge the individual article’s or author’s performance. Even under the assumption 
that citations were equally distributed among all articles, the IF w ould only measure the 
interests of other researchers in an article, but not the article’s importance and usefulness. 


two previous years (e.g. 1997-1998) were cited in a 
particular year (e.g. 1999)”. By accepting the impact 
factor as a measure of quality, a prevalent view 
among many groups and institutions 2 (see the 
criticism by Kokko & Sutherland 1999), a message is 
sent that articles published in high impact journals 
are superior to those published in low impact journals 
or a journal with no impact at all (see the discussion 
in Leimu & Koricheva 2005). However, we must clarify 
that this is just one of the understandings on the IF. 
Other researchers assume that the IF is a measure of 
visibility of a publication, and quality and visibility 
certainly not always go together. While not going 
together, implications of terms in practice are quite 
different. In this article, I assume that IF is a measure 
of visibility, which many researchers and institutions 
take as a measure of quality (see notes below). Thus, 
every time we refer to the IF, we will be giving the 
visibility connotation. Box 1 presents a discussion 
of questions the IF can help answer. 

If we then assume that citations constitute an 
important and robust variable in the evaluation of 
science, we should also assume that these citations 


About this discussion, according to Figa-Talamanca (2007:86): 'The impact factor 
is undoubtedly vety strong. It succeeded in imposing itself as a measure of quality, 
by simply asserting its value. This assertion cannot be disproved, and, as long as it 
is taken for granted, it produces effects. The strength of impact factor is proved 
beyond any doubt by the elects it produced in the scientifc environment” (emphasis 
ours) The notion of using the impact factor as a measure of quality is not an Isolated 
one. and it has very old roots (see the quotes and comments in Falagas etal. (2006)). 


are not influenced by other factors or variables (see 
Fig. 1). If other factors do have an influence over 
the number of times that a work is cited, we must 
necessarily acknowledge that the Impact Factor is 
not a good measure. Thus, this article aims to 
reinvigorate the discussion of how citations are 
used in the calculation of the Impact Factor (IF) 
and to show how the number of citations of a work 
seems to be a not good variable (in the way it is used 
today) to stratify scientific production and scientists. 
While it is understood that citations strongly 
influence calculations of the IF, it is not assumed 
here that the two are equivalent. It should be noted 
that this topic is very complex and controversial; 
my arguments will be developed using specific case 
studies drawn from the biological sciences 
(especially from ecology). Moreover, the IF is 
susceptible to several factors that may not be 
related to citations. 

Are citations influenced by the number 
of authors of an article or by gender? 

We define publication bias as the situation in 
which the merit of a manuscript is evaluated using 
factors tangent to its intrinsic characteristics as a 
scientific work (Fig. 1). It has been observ ed that 
male authors attract more citations than female 
authors (Cassey & Blackburn 2003, 2004). This 
relationship was shown to be insignificant in the 
works of Leimu & Koricheva (2005) and Borsuk et 
ul. (2009) in an analysis of publications in the field 
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Figure 1 - A conceptual illustration of the publication and dissemination process in ecology with potential publication 
related biases listed to the right. Attributes at each level of the process are listed as within study (i.e. attributes of the study 
itself and subsequent interpretations by the author), publication level (properties of the published study), and journal level 
(associated attributes of the Journal where in a given study is published). Citation frequency refers to the number of times 
a particular publication is Referenced in other publications; however, the impact factor attribute of a journal is also 
calculated using mean citation frequency of all papers published there in within the last two years and also best conceptualized 
as an emergent property. The biases listed are generally recognized within ecology and sometimes quantitatively analyzed 
and published. Nonetheless, few analyses have explored either the potential relationships between attributes of the publication 
process, as denoted by the dotted arrows, or between the attributes of the process and the various biases which may 
influence relative success or performance of a study/publication, as denoted by the double-headed arrows. Source: Lortie 
etal. (2007:1250). Reproduced with the permission of the publisher. John Wiley and Sons (License number: 2482660790074). 


of ecology. The author’s gender also does not seem 
to affect the rejection rates of articles (Aarssen et 
al. 2008). Nonetheless, Tregenza (2002) reached a 
conclusion that is quite interesting: there are 
differences related to gender in the acceptance rate 
of works that would be difficult to explain without 
accepting the idea of bias. 

A significant relationship has been found 
between the number of authors of an article and 
the citation rate (see Tregenza 2002). This suggests, 
for example, that when there are more authors on 
an article the possibilities of self-citation also 
increase (Borsuk et al. 2009). However, there are 
measures that may be used to avoid bias caused by 
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the self-citation effect 3 . Borsuk et al. (2009) suggest 
that the number of authors could reflect: a) the external 
perception that multi-authored works have better 
quality, as they contain the collaboration and vision 
of different people, or b) that the scientific community 
“perceives” greater merit in the work; after all, "how 
many people could be deceived?" 


’When 1 finished writing this article, a strong debate about the high Impact Factor 
that a Brazilian journal achieved in 20OT (3.46) started to circulate in the internet 
There were some discussions that this value was only reached because of the high 
number of sclf-eitations incentivized by the editors. This episode only shows the 
great fragility ofthe IF, how it ean be easily manipulated and this was not a unique 
episode: sclf-eitations arc current practices in many internalional journals. So. 
why do we still wont to be subjugated by this failure system? 
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Regardless of the possible explanations, it is 
evident that collaboration in scientific works brings 
many benefits, such as those cited above. Lcimu & 
Koricheva (2005) add that this relation increases 
when the number of authors exceeds four. Curiously, 
these authors also found that interdisciplinary 
collaboration enhances the citation rates of 
ecological studies, though this is not true if the 
collaboration involves different departments within 
the same institution. Also, in studies on the field of 
ecology, an interesting article by Lcimu & Koricheva 
(2005) states that a positive relation exists between 
self-citation and external citation. 

In these times of economical crisis, there is 
an increasing pressure on scientists to increase 
their publication output, and misconduct, such as 
including authors who did not directly participate 
in the production of an article, has become the norm. 
For more on the consequences of this pressure to 
increase one’s number of publications, see 
Martinson el al. (2005) and Albuquerque (2009). 

Are citations influenced by the 
name of the author or his language? 

According to Tregenza (1997), “Darwin may 
be a better name than Wallace!” Tregenza suggests 
that a significant relationship exists between citations 
and the initial of the last name of the author in a 
scientific publication. However, Lcimu & Koricheva 
(2005) were not able to reproduce these findings in 
relation to the alphabetical order of the last names 
of authors in ecology. These contradictory results 
suggest that there is a need for new evaluations in 
other areas of research. 

Authors whose native language is English 
attract more citations than authors whose native 
language is not English, even when publishing in 
English (Leimu & Koricheva 2005). This assertion has 
been reinforced by the findings of Borsuk et al. (2009). 
In addition, this factor appears to affect not only the 
citation rates but also the probability of an article being 
accepted in ecology journals (Tregenza 2002)° 
Tregenza expresses surprise at this pattern when he 
states that Patterns of acceptance relative to the 
country of author affiliation are intriguing and might 
indicate that non-English speaking nations are at a 
disadvantage either as a result of discrimination or 
perhaps because of the additional burden of having 
to write in a foreign language”, (p. 350). 

One could argue that this might reflect a higher 
requirement on the part of journals to accept works 
written in higher quality English. Considering that most 


journals that have been classified by CAPES as 
possessing impact are not national, could this possible 
bias not be an impeditive factor that hampers the 
publication of Brazilian researchers? Mencghini etal. 
(2008) obtained a disturbing finding: there is a tendency 
for Latin American authors to be less cited in journals 
with high impact factors, except when there is 
international collaboration. Although the authors 
argue that it is currently difficult to define the reasons 
for this phenomenon, they do suggest two possible 
reasons: psychosocial source bias (discrimination) or 
real differences in the scientific visibility of these 
articles. An anonymous reviewer, who was 
commenting on this paragraph, highlighted important 
issues that also deserve our consideration in the 
statement: Has this to do with bias, or has to do with 
the fact of science in Latin America not having 
historically concern about its international visibility? 
In other words, is it guilty of a Machiavellian system 
or is it simply our historic failure?” (p. 4). 

Final considerations 

The aforementioned examples, based on 
systematic studies, were highlighted to engender 
debate without passing premature judgment. 
However, the writer’s inclination is to be severely 
critical of applications of the IF. It has become 
evident that evaluations based on the IF may suffer 
from a strong bias, causing research to be judged 
not solely on its scientific merit but also on 
attributes of the authors, nationality, the existence 
of cooperation, etc. (see Lortie etal. 2007). 

With this assumption, we should necessarily 
push for debate and discussion of the validity of the 
IF. Moreover, I believe that a culture of reflection and 
self-criticism should be encouraged. It is clear that 
scientists publish their results to be read, cited, and 
remembered (Abt 1998); however, the discussion 
herein shows^that accomplishing these goals, as 
measured by the Impact Factor, depends upon more 
than the quality or visibility of the work. Abt (1998) 
states what many people already know for certain: the 
number of citations is influenced by the field of 
knowledge (velocity and growth) and even the length 
of the research itself! This debate is obviously very 
rich as well as conducive to the enrichment of scientists 
knowledge. To further illustrate the complexity of these 
issues, consider the statement by Simkin & 
Roychowdhury (2003) that many authors categorically 
do not read the papers they cite. They estimated that 
only 20% of authors have read the work they cite. 
They are several motives (psychological, sociological, 
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political, historical, etc.) influencing an author’s 
decision to cite a study, and this is an object of ongoing 
interest by scholars in various fields (see Brambilla et 
al. 2006). The latter authors reached this interesting 
conclusion: “It was found through content analysis, 
that all works cited an article for the same reason: a 
description of theoretical and practical methods to 
solve problems. However, subjective reasons 
permeated the act of quoting, and only a more accurate 
analysis showed such reasons. It was observed that 
the article was also cited because their authors share 
a common paradigm and build expertise in network 
and collectively. This could be verified because the 
citing articles were written in co-authored by the group, 
in collaboration with researchers from other Brazilian 
and foreign institutions, and were published in 
national and international journals with importance in 
the area) “(p. 206). 

Thus, it becomes more evident the complexity 
of the issue and the need to treat it with care and 
seriousness. Finally, to conclude my discussion on 
the IF, Colquhoun (2003) categorically states that: 
“Eugene Garfield, who invented the wrecheted 
impact factor, himself said that it is not appropriate 
for ranking individuals (...). Astonishingly, these 
facts are not known (or are ignored) by some 
selection committees * 4 5 .’’ (p. 479). 

Colquhoun (2003) has a curious example of 
having published a work in Nature (with an impact 
factor at the time of 27.9) and having only been cited 
57 times, while another work published in a journal 
of less impact (3,1) attained more than 400 citations! 
This indicates that the publication of a work in a high 
impact journal is not a guarantee that the work will 
achieve the much-desired dream of the author: to be 
read, cited, and remembered. The data of the ISI (The 
Institute of Scientific Information), for Nature in 2001, 
reinforce this. In 1999, of the 858 works published, 
the data show that the 80 works most cited comprised 
half of all citations! (Colquhoun 2003). This finding is 
not new, rather it is typical. As a result, many began 
to adopt the h- index 3 as a measure of authors’ 


*Wtulc this fact may be known by many development agencies in the world, the 

criticism is valid because many people remain unaware. 

5 The A-index was proposed by Mirsch (2005:16569). Thus the author stands on its 
proposal: ’This large amount of information will be evaluated with different criteria 
by different people. Here, 1 w ould like to propose a single number, the “A index,” 
as a particularly simple and useful way to characterize the scientific output of a 
researcher. A scientist has index A if A onus or her Np papers have at least h citations 
each and the other (Np - A) papers have <*A citations each”. Some criticisms were 
made use of this measure to assess “the cumulative impact and relevance of an 
individual’s scientific research output” (Hirsch, 2005:16569). See Zhang (2009) 
for criticisms and suggestions for improvement of the A-index. 
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visibility. According to Scarano (2008):“While 
evoking the need of better ways of assessing 
scientific literature, the editors admitted they would 
be lying if they said they were not interested in their 
journal’s impact factor. This exemplifies that despite 
one’s awareness of the limitations and fiawsofimpact 
factor judgement, it is simultaneously the most 
practical way of producing any type of objective 
analysis of scientific value.’’ (p. 191), 

I have many doubts about the statement 
above. Everyone can agree that objective and fair 
evaluation criteria are required. The limits of any 
measure should also be widely appreciated to avoid 
any potential inconsistencies. While I would 
sincerely like to end this text with a proposal that 
mitigates the shortcomings discussed, I will 
conclude by repeating the topical question, “What 
will be then the best existing alternative?’’ 6 and by 
suggesting that this issue should be discussed 
more openly within the scientific community. 
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